BiPO

BiPO (Bi-directional Preference Optimization) is starting with a zero-initialized steering vector that has learnable parameters (requires_grad=True) and training it through gradient descent optimization rather than simple averaging. During training, it feeds positive and negative activation pairs through a preference loss function that encourages the steering vector to increase the model's preference for positive examples over negative ones, updating the vector through backpropagation for a specified number of epochs (default 100).

The loss function computes preference scores by taking dot products between activations and the steering vector, then uses a margin-based loss to optimize the vector. During inference, the trained vector is applied exactly like CAA by adding it to activations at the target layer with a strength multiplier, targeting the second-to-last token position.

This learned approach can potentially capture more complex patterns than simple averaging but requires more computational time for training.