transform the problem

Given two points defining a "centerline" and an arbitrary third point, find the distance from the point to the line.

Wikipedia has a page on this problem which provides derivations of the below solution using algebraic, geometric, and vector-based approaches. The distance \(d\) from the point \(\left( x_0,y_0 \right)\) to the line \(ax + by + c = 0\) is \[ d = \frac{\left\vert ax_0 + by_0 + c \right\vert}{\sqrt{ a^2 + b^2 }} \] But what if you need to know which side of the line the point is on? As an example application, I need to tell a vehicle which direction to steer in order to stay on the line defined by some start point and end point. In the vector proof, they offer that "although the distance is given as a modulus, the sign can be useful to determine which side of the line the point is on, in a sense determined by the direction of normal vector \(\langle a,b \rangle \)". The problem is that the normal vector \(\langle a,b \rangle \) is the same if you switch the start and end points used to find \(a\) and \(b\) in the first place. (The equation of the line is the same, regardless of which two points are used to define it.)

To get the sign of the distance, translate the three points so that the start point is at the origin, then rotate so that the end point lies on the x-axis. The distance is the y-coordinate of the transformed point.

Let \(Q\) and \(R\) be the start and end points, respectively, and \(P\) be the position of the vehicle. The subscript \(t\) will denote translation. For the translation, simply subtract \(Q\)'s coordinates from all points. \[\displaylines{ Q_t = \left( Q_x - Q_x, Q_y - Q_y \right) = \left( 0, 0 \right) \\ R_t = \left( R_x - Q_x, R_y - Q_y \right) \\ P_t = \left( P_x - Q_x, P_y - Q_y \right) }\]

For the rotation, the angle of \( \vec{R} \) (\(\phi\)) will be subtracted from that of \(\vec{P} \) (\(\theta\)) (now treating the points as vectors). Since the rectangular coordinates of the points are given, use the tangent: \[ \displaylines{ \tan{\phi} = \frac{ R_{t,y} }{ R_{t,x} } \\ \tan{\theta} = \frac{ P_{t,y} }{ P_{t,x} } }\]

The typical problem with tangent's loss of information will be accounted for by adding \(\pi\) to the angles of points that lie outside its domain (points with a negative x-coordinate). The issue here is that arctangent will only give angles in the range \(\left[-\frac{\pi}{2}, \frac{\pi}{2} \right] \), but the full circle \(\left[0, 2\pi \right] \) is needed. With the adjustment: \[ \displaylines{ \phi = \begin{cases} \arctan{ \left( R_{t,y}/R_{t,x} \right) } \\ \arctan{ \left( R_{t,y}/R_{t,x} \right) } + \pi & \text{if $R_{t,x}$ negative} \end{cases}\\ \theta = \begin{cases} \arctan{ \left( P_{t,y}/P_{t,x} \right) } \\ \arctan{ \left( P_{t,y}/P_{t,x} \right) } + \pi & \text{if $P_{t,x}$ negative} \end{cases} }\]

After these transformations, the geometry yields an easy calculation of the distance from point \(P\) to the line \(QR\) — simply the y-coordinate of \(P\) — and the sign of that number consistenly indicates whether \(P\) lies on the "right" or "left" side of the line, defined by the order of \(Q\) and \(R\).