I'm gonna try this again

I've been trying to setup a simple camera class with OpenGL but I'm completely lost and I've made zero progress creating anything useful. I'm using modern OpenGL and the glm library for matrix math. To get the most basic thing I can think of down, I'd like to pan an arbitrarily positioned camera around. That means move it along its own Up and Side axes. Here's a picture of a randomly positioned camera looking at an object:

It should be clear what the Up (Green) and Side (Red) vectors on the camera are. Even though the picture shows otherwise, assume that the Model matrix is just the identity matrix. Here's what I do to try and get it to work:

Step 1: Create my View/Camera matrix (going to refer to it as the View matrix from now on) using glm::lookAt().

Step 2: Capture mouse X and Y positions.

Step 3: Create a translation matrix mapping changes in the X mouse position to the camera's Side vector, and mapping changes in the Y mouse position to the camera's Up vector. I get the Side vector from the first column of the View matrix. I get the Up vector from the second column of the View matrix.

Step 4: Apply the translation: viewMatrix = glm::translate(viewMatrix,translationVector);

But this doesn't work. I see that the mouse movement is mapped to some kind of perpendicular axes, but they're definitely not moving as you'd expect with respect to the camera. Could someone please explain what I'm doing wrong and point me in the right direction with this camera stuff?