Tuesday, August 09, 2011

Dates

2nd-15th September - San Sebastian, Spain

Monday, August 01, 2011

FACET Experimental Installation

When I interviewed for this job way back last September, I was petrified. Not by the interview which was a relaxed affair in the corner of the DWB common room, but by the description of the job. I remember walking up Longwall Street to the Nat Hist Museum (I was meeting with skit and told her to meet me in the graveyard to St Giles but then suddenly phoned her to tell her to meet me by the fossilised tree exhibit instead- I like dead things) and talking to my mum on the phone about how scary it sounded. And then that night I barely slept, it was just such an awful prospect.

The job was to build a facility. Essentially from scratch. There is a linac at SLAC capable to delivering beams but that's all pretty useless without experiments to deliver them to. The experiments were all lined up and waiting but the infrastructure (cables, computers, controls and other words that don't necessarily start with the letter c) was not.

So I was scared. And this made me say yes.

I wouldn't class myself as a risk taker but at certain points in my life, I have put myself out there and not (yet) regretted it.

Building FACET (the new SLAC facility) wasn't exactly what I thought it would be. Much of what I had to do was there. People had dropped cables between the tunnel and a rack in the klystron gallery. Unfortunately, it wasn't really very useful there so more cables needed to be routed from the rack to where they would be used. Which was a trailer, on order and yet not yet at SLAC. Beamline parts were needed, procedures had to be documented, safety reviews needed to be done, users needed to be registered and trained, controls needed to be developed...

And in the past couple of weeks, everything needed to be installed and working.

I was given 8 days to pull everything together. In the end, only one of the five experiments got installed. Two others were almost there. Two just weren't anywhere near ready enough. At least one of those two had the good grace to inform me in advance but one kept me in the dark until five days before the installation about not being ready leading to a last minute rush on getting something else to install instead.

Somehow everything was going to plan except for cable drops and networks. The person I was assigning the organisation of the cable work to was just not delivering. I have yet to complain to his boss but I ought to. The networks was just horrific as the person that designed it for us had gone off on holiday without implementing it or sharing the details with his colleagues. This led to an absolutely atrocious state of things where not only did we not have networks set up, we also had a group of people angry at us for being cowboys as from their perspective, there was no plan at all. To further complicate things, one experimenter that promised a stand alone controls system suddenly brought a system that could only run on a certain network for which we didn't have the infrastructure to support.

Anyway. Everything was more or less going to plan except that Uli, the head of the department that runs the linac, wanted us to end two days early to give him extra time to tune the beam. I agreed to this and there were some fireworks that came from that (since Uli's definition of two days early was actually 2.5 days early) but basically, it looked like we could do it.

Until Saturday when we found a leak in the beamline.

The leak was between two areas separated by a beryllium window. A leak isn't usually a worry but in this case, there was a possibility that the leak was due to a break in the beryllium. It seemed remote but, worryingly, the beryllium window came with a side labelled "vacuum" and, well, our vacuum was on the other side. I was distracted that Saturday by trying to solve the network issue (a lost cause but I didn't know that yet) and not down in the tunnel. I left instructions on which volume to pump on first. They were not followed and I still don't know why. So in realising that the pump down procedure had not been adhered to plus the leak of unknown origin, we went into panic mode.

We wrote emails to our supervisors and to the safety group. We halted the pump down. And so we lost days of time.

Nothing could be done until Monday. So that lost two days of the schedule (and of course, Uli did not get his extra time). And on the Monday, things just stopped. The window was removed and everything was safe but we weren't even able to inspect it- it was bagged and that was that. I asked for the window to be inspected and leak tested and that just garnered a whirlwind of trouble as there was no procedure for that. And then it was discovered that there was no written procedure for anything that we had been doing. We discussed it but we did not write it.

It took most of the day to even realise why we weren't working. I spoke to the supervisor for the technicians and everything seemed fine but he still didn't allow them to work. In the end, I had to suffer a long meeting of being asked awful questions about what I was doing and who was I to be doing it, what authority did I have and what actual consideration had I put into doing it.

Being asked these questions was just awful. I've been doing my best but I'm not an engineer or at all experienced in the procedures at SLAC. I was just trying my best to get the facility built.

The days leading up to this were hectic. I was living life on a carousel and juggling with people on the outside. Throwing the right balls to the right people and trying not to fumble took so much attention. I barely paused, I barely breathed. And on Monday I just felt like a hopeless failure.

On Tuesday, we continued with the work having written and agreed to a procedure. We put a new window in and pumped it down. And continued pumping and continued pumping. Our vacuum was shot. One side of the window was too bad for even the ion pumps to come on. The other side was better but critical because the gauge readings were interlocked to the valves and we needed to set higher values just to get the valves to stay open.

We pumped overnight. It was then Wednesday and things looked only slightly better.

I rustled a couple of vacuum engineers to look at our system to recommend improvements. We've not had a chance to plan yet. The past couple of days have really been needed to recover our energy levels. But we really need to plan again. We need to go back in and tear it apart to build something that can actually be pumped down.

The horror is not yet over.

Was it at all worth it? I signed up for the job knowing that this would make or break me. I need more perspective to tell.