Open the slide deck (PDF)

The video above is a keynote speech delivered at a programming languages (PL) conference in 1998. In essence, the speech argues that programming language design should move toward providing meta-level tools that allow users (developers) to extend the language themselves and build their own libraries. In other words, a “cathedral model,” in which a small group of designers centrally drives language design, tends to alienate users and respond slowly to change. Instead, a model closer to that of the Linux operating system — where users actively participate, rapidly modify, and evolve the system — is far more effective.

The core message of the talk lies in this growth model of programming languages. As concrete examples of such an approach, the speaker mentions generic types in Java as a pattern for defining types, operator overloading that allows user-defined types to naturally use operations such as + and -, and lightweight types that can be stored in structures like stacks.

From this perspective, the key challenge in programming language design is how to incorporate users (developers) into the process of language evolution. Rather than language designers attempting to build a large, complete language upfront, they should design tools that enable developers to extend the language to solve problems specific to their own projects. Observing that the vision proposed by the speaker was later adopted in Java and reflected in other languages such as C#, one can see that this philosophy of language design is not only idealistic but also practically effective.

Moreover, this philosophy extends beyond programming language design and offers insights into the design of complex systems in general. Many modern systems, programming languages, and libraries are developed as open-source projects. This can be interpreted as a conscious effort to include users in the development process. Of course, languages developed as open-source projects (such as Python) also have clear drawbacks, including difficulties in maintaining a fully consistent syntax. Addressing these issues through improved development processes remains an important challenge. Nevertheless, this speech presents a compelling direction for how large-scale, complex projects can be built into successful systems that are widely used by diverse communities.

The fundamental challenge in solving MDPs is that the Bellman Equation — which defines the value of a state or action — creates a massive system of simultaneous equations. For a problem with S states and A actions, solving this system directly is computationally infeasible.

DP circumvents this by using iterative updates and the concept of bootstrapping (where updates to the value function are based on estimated values of successor states). This approach falls into two main categories: Policy Iteration and Value Iteration.

1. The Two Goals: Policy Evaluation and Policy Control

DP algorithms are built around two interlocking sub-problems:

1. Policy Evaluation: Given a specific policy pi, calculate the state-value function $V_{\pi}(s)$ (the expected long-term return from state s if policy pi is followed). This is an intermediate step.
2. Policy Control: Find the optimal policy pi* that maximizes the value function V*(s) for all states. This is the ultimate goal.

2. Policy Iteration: Evaluate, then Improve

The Policy Iteration method is a complete solution that alternates between Policy Evaluation and Policy Improvement until the optimal policy is found.

Step 1: Policy Evaluation (Iterative Policy Evaluation)

This step calculates the value function $V_{\pi}$ for the current policy pi. It uses the Bellman Expectation Equation iteratively:

Process: Starting with an arbitrary value function $V_0(s)$ (often all zeros) , the value of each state $s$ is updated based on the estimated value of the next state s’ (bootstrapping).

• Convergence: This process is repeated until the maximum change in the value function ($\Delta$) across all states drops below a small threshold. The value function $V_k$ is guaranteed to converge to the true $V_{\pi}$ under the given policy $\pi$.
• Update Style: A “in-place” update (where the current state’s new value is used immediately to calculate the next state’s value, known as the “overwrite method”) is generally faster than using a separate new_V table.

Step 2: Policy Improvement (Greedy Policy)

Once $V_{\pi}$ is accurately calculated, the policy is improved by taking an action that is greedy with respect to the new value function.

• Greedy Action: For each state $s$, the new policy $\pi’(s)$ chooses the action $a$ that leads to the best combination of immediate reward ($R$) and discounted value of the next state ($\gamma V_{\pi}(s’)$).
• Policy Improvement Theorem: This theorem guarantees that the new greedy policy $\pi’$ will be better than or equal to the previous policy $\pi$. If $\pi’$ is strictly better, the process repeats.

Convergence to Optimal Policy

Policy iteration alternates between evaluation and improvement. Since there is a finite number of policies in a finite MDP, this process is guaranteed to converge to the optimal policy pi* and its optimal value function V*.

3. Value Iteration: Combining Evaluation and Improvement

Value Iteration is a simplified DP algorithm that implicitly combines the Policy Evaluation and Policy Improvement steps. It does not require a full, separate policy evaluation phase.

The Core Idea: Bellman Optimality Equation

Instead of iteratively solving the Bellman Expectation Equation (Policy Evaluation), Value Iteration directly uses the Bellman Optimality Equation for V*:

• Single Step: In each iteration ($k \to k+1$), the value of state $V(s)$ is updated by taking the maximum expected value over all possible actions $a$. This $\max_{a}$ operation is the policy improvement step.
• Evaluation and Improvement: By taking the $\max_{a}$, the algorithm effectively performs a one-step greedy policy improvement at the same time it’s updating the value function (evaluation).

Convergence and Final Policy

• Convergence: Like policy evaluation, the value function $V_k$ is guaranteed to converge to the optimal value function V*.
• Extracting pi*: Once V*is found, the optimal policy pi* can be extracted in a single step by performing a final greedy policy operation with respect to the converged V*.

Policy Iteration vs. Value Iteration (Generalized Policy Iteration)

Both Policy Iteration and Value Iteration are forms of Generalized Policy Iteration (GPI), where Policy Evaluation ($V \to V$) and Policy Improvement ($\mu \to \mu’$) interact until convergence.

• Policy Iteration performs a full evaluation (many V updates) before a single improvement (one pi update).
• Value Iteration performs a one-step improvement for every state update (no need for the separate, time-consuming full evaluation).

Value Iteration typically converges faster by reducing the number of evaluation steps, making it computationally more efficient, especially in large MDPs.

<hr><p>Dynamic Programming in RL: Policy Iteration vs. Value Iteration 🚀 was originally published in KAIST Include AI Club on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>

But what exactly does “optimal” mean, and how do we mathematically model this continuous interaction? The answer lies in the Markov Decision Process (MDP) and the fundamental Bellman Equation.

1. Modeling the RL Problem: The Markov Decision Process (MDP)

In RL, the agent interacts with an Environment. The agent perceives the environment’s state using a Sensor (percept/observed state) , decides on an Action using a Function (Policy) , and executes it via an Actuator. The environment, in turn, provides a Reward.

The MDP provides the formal, mathematical framework for this interaction:

State: Current description of the environment.

Action: Possible moves the agent can take.

Policy: The agent’s strategy for choosing an action.

Transition: How the environment’s state changes.

Reward: Evaluation of an action (e.g., cost).

The ultimate goal of RL is to learn the policy ($\pi$) that maximizes the accumulated Reward.

The Importance of the Markov Property

A key assumption in the MDP is the Markov Property: the future state s’ only depends on the current state ($s$) and action ($a$).

In a Markovian State Transition Model, the probability of the next state is defined as $P(s’|s, a)$. This property is vital because it allows for a simplified, powerful recursive solution (the Bellman Equation) that doesn’t require considering the entire history of states.

2. Measuring Value: State Value Function

In the Multi-armed Bandit problem (a simplified RL problem without state transitions), the quality of an action Q is the expected reward from that action.

However, in the full MDP, the reward r doesn’t just depend on the action; it depends on the resulting state transition (S, a, s’). Therefore, we need a more powerful way to measure value.

Return and Discount Rate

We define the Return as the total weighted sum of future rewards.

The Discount Rate (in [0, 1)) is a hyperparameter that determines the importance of future rewards compared to immediate rewards.

State Value Function

The State Value Function is the expected return (G_t) starting from state $s$ while following a specific policy pi:

This function is critical because it allows us to compare policies. An optimal policy pi* is one that maximizes $v_{\pi}(s)$ for all states s in S. The Deterministic Optimal Policy Theorem states that for any finite MDP, at least one deterministic optimal stationary policy pi* exists.

3. The Bellman Equation: The Recursive Definition of Value

The Bellman Equation is derived from the MDP and is valid for all MDP problems. It is important because it shows the recursive definition of the value function.

Recursive Definition of $v_{\pi}(s)$

The magic of the Bellman Equation lies in rewriting the infinite sum of the Return G_t into a simple recursive expression:

Applying the expected value based on the policy pi and the transition T, the State Value Function $v_{\pi}(s)$ can be expressed as:

In simple terms, the value of the current state ($v_{\pi}(s)$) is the expected value of the immediate reward plus the discounted value of the next state ($v_{\pi}(s’)$). This allows us to deal with the value function without an infinite number of terms.

Recursive Definition of Action Value Function $q_{\pi}(s, a)$

The Action Value Function ($q_{\pi}(s, a)$) measures the expected return when starting in state $s$, taking action $a$, and then following policy $\pi$.

It is also defined recursively and is related to $v_{\pi}(s)$:

The Bellman Optimality Equation

The most important insight is the Optimal Bellman Equation, which defines the conditions that the optimal value function ($v_*$ or $q_*$) must satisfy.

The optimal policy ($\pi^*$) always chooses the action with the highest value.

1. Optimal State Value: The optimal value of state $s$ is the maximum expected value over all possible actions $a$:

2. Optimal Action Value: The optimal value of state-action pair $(s, a)$ is the expected immediate reward plus the discounted maximum optimal action value in the next state $s’$38:

The optimal policy ($\mu^*$) is then simply the action that maximizes this optimal action value:

The Bellman Optimality Equation serves as a critical target function condition for solving RL problems41. Any policy, value function, or action-value function that models an MDP must obey the Bellman Equation.

<hr><p>The Foundation of Reinforcement Learning: MDPs and the Bellman Equation 🧠 was originally published in KAIST Include AI Club on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>

1. Defining GNN Expressiveness: The WL Test Limit

Expressiveness is the metric that measures how well a GNN can distinguish between different graph structures. The primary goal of a GNN is to generate similar embeddings for similar nodes and different embeddings for different nodes.

The gold standard for determining a GNN’s distinguishing power is its connection to the Graph Isomorphism Problem , which asks whether two graphs have the same connectivity structure.

The Weisfeiler-Leman (WL) Test

The Weisfeiler-Leman (WL) Test is an algorithm designed to address the graph isomorphism problem and is considered the theoretical upper bound for the expressive power of many GNNs.

How the WL Test Works:

1. Initial Coloring: All nodes are initially assigned the same color.
2. Color Aggregation: Each node collects and aggregates the colors of itself and its neighbors (e.g., using a Sum Aggregator).
3. New Color Generation: The aggregated result is input into a hash function to generate a new color.
4. Iteration: Steps 2 and 3 are repeated until the node colors no longer change.

If two graphs do not share the same final set of colors, they are considered non-isomorphic. The WL test can distinguish more graph structures than other methods.

Graph Isomorphism Network (GIN)

The Graph Isomorphism Network (GIN) is a GNN architecture designed to have the same expressive power as the WL test. Its aggregation and combination steps mimic the iterative coloring of the WL test:

Graph Classification and Readout

Graph Classification is the task of predicting a property of an entire graph by transforming the node embeddings produced by a GNN into a single graph embedding. This is done using a Global Pooling or Readout function.

2. Link Prediction: Forecasting Future Connections

Link Prediction is the problem of predicting whether a link (edge) exists or will exist between two nodes.

Traditional Link Prediction Methods

Traditional approaches rely on heuristics and matrix decomposition:

Heuristic Methods

Local Heuristics (1-hop or 2-hop):

• Common Neighbors: Measures the number of neighbors shared by two nodes.
• Jaccard’s Coefficient: Measures the proportion of common neighbors.
• Adamic-Adar Index: Sums the inverse of the log of the degree of common neighbors, assigning less importance to common neighbors with many connections.

Global Heuristics:

• Katz Index: Calculates the weighted sum of all possible paths between two nodes, penalizing longer paths.

Matrix Factorization (MF)

This method learns node embeddings Z to approximate the entire adjacency matrix A. The goal is to maximize the inner product of embeddings for similar nodes and minimize it for dissimilar node.

GNN-based Link Prediction Models

Graph Autoencoder (GAE)

The GAE uses a GNN (e.g., a two-layer GCN) as an Encoder to generate node embeddings .

A Decoder then approximates the adjacency matrix A hat using a sigmoid function on the inner product of the embeddings:

Variational Graph Autoencoder (VGAE)

VGAE is a probabilistic variant of GAE. Instead of learning Z directly, it learns the parameters (mean and variance) of a latent normal distribution that Z follows. Embeddings are sampled from this learned distribution, and the loss function includes the Kullback-Leibler (KL) Divergence term to ensure the output distribution is close to a prior distribution.

SEAL (Subgraphs, Embeddings, and Attributes for Link prediction)

SEAL focuses on local connectivity by using enclosing subgraphs as input instead of the entire graph. This approach automatically learns local heuristics.

• Enclosing Subgraph Extraction: Lists the target nodes and their k-hop neighbors and extracts their edges and features.
• Node Labeling (DRNL): The Double-Radius Node Labeling (DRNL) algorithm generates a single integer label that captures the structural importance of each node by encoding its distances from both target nodes x and y.

3. Graph Generation: Creating New Network Structures

Graph Generation models are crucial for applications like data augmentation, anomaly detection, and drug discovery. They can be categorized into realistic generation (mimicking existing graphs) and goal-directed generation (optimizing a specific metric).

Traditional Graph Generation Methods

• Erdos-Renyi Model (G(n, p) or G(n, M)): Assumes each edge exists independently with probability p or chooses a graph with M edges randomly. A limitation is the difficulty in mimicking real-world graphs with clusters or communities due to the independence assumption.
• Small-World Model (Watts-Strogatz Model): Captures the real-world features of short path length and high clustering coefficient. It starts with a regular lattice and randomly rewires some edges with probability p to introduce shortcuts. Its limitations include a fixed number of nodes and difficulty creating realistic degree distributions.

GNN-based Deep Graph Generation Models

Variational Autoencoder (VAE) Models

VGAE can be used for generation: it stores the structural information of the original graph in a probabilistic distribution45. A new graph is created by sampling from this distribution and setting a threshold to create a binary adjacency matrix.

• Constrained Graph Variational Autoencoder (CGVAE): Extends VGAE to learn more complex graphs G=(A, E, F) that include the adjacency tensor A, edge attribute tensor E, and node feature matrix F. It can generate graphs with specific conditions (e.g., valid molecular structures) by adding constraints or regularization.

Autoregressive Models

These models view graph generation as a sequential decision-making process. At each step, the model creates a new node or link, and the resulting partial graph becomes the input for the next step.

GraphRNN: Uses two Recurrent Neural Networks (RNNs):

1. Graph-level RNN: Generates a new node and sets its initial state52.
2. Edge-level RNN: Predicts the probability of the new node connecting to existing nodes, filling in the adjacency matrix row by row53.

Generative Adversarial Networks (GANs)

GANs use a zero-sum game framework where a Generator and a Discriminator compete.

• Generator: Creates new graph data that resembles the training data.
• Discriminator: Distinguishes whether a sample is a real graph or a fake one created by the generator.
• MoIGAN (molecular GAN): Combines the Wasserstein GAN (WGAN), which improves training stability, with a Reinforcement Learning (RL) objective to generate molecules with desired chemical properties.
• Generator: Outputs the Adjacency tensor (A, edge/bond types) and the Annotation matrix (X, node/atom types).
• Reward Network: Scores each generated graph based on a real-world score (e.g., from RDKit), providing a reward for generating a molecule that meets the desired chemical characteristics.

Writer: MinJae Choi

<hr><p>Unlocking the Power of Networks: Expressiveness, Link Prediction, and Generation in Graph AI was originally published in KAIST Include AI Club on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>

Open the slide deck (PDF)