Status of the 2018 Refurbishment of the Crocker Nuclear Laboratory Cyclotron

Updated September 11, 2018:

Recent status updates:

September 11th, 2018:
- Beam came up with yesterday’s setting, so we’re officially saying we’re back in business.
September 10th, 2018:
- Success! We were able to extract full energy beam this morning. Randy is working to increase the current.
  
  Big thanks to Randy, Tim, Brian, Hans, Michael, and Carlos for working so hard. I also want to thank Jim Ball, from our former sister cyclotron at Oak Ridge, who has been doing a lot of simulations for us, which convinced is that the outer trim coil settings really could be the root of the weird behaviors we were seeing.
September 7th, 2018:
- 11:37AM: I apologize for the dearth of updates. We’ve been very frustrated trying to extract beam at full energy. It appears that the orbit had shifted and the beam was hitting on the other side of the orbit before getting to the septum. This morning, we had a bit of a breakthrough and for the first time, we are able to get to the full radius. Randy is trying to tune for extraction. Stay tuned.
August 24th, 2018:
- 11:35AM: Spent yesterday checking everything. No big problems found, but lots of minor alignment issues, which have been fixed. We verified the symmetry and polarity of all 10 trim coils, but Tim discovered that the blocking capacitors are bad on all the outer trim supplies (this apparently can happen when they’re off for a long time). This would cause excessive ripple, which could be the source of our problems. He’s replacing these, and we are cleaning up and reinstalling the ESD. We should be pumping by this afternoon.
August 22nd, 2018:
- 10:17AM: Opened up and removed ESD. No obvious problems. I’ve been in communication with several ORIC people (identical cyclotron that was at Oak Ridge. They pointed out that beam is getting lost at the boundary between two trim coils and they had a similar problem when a trim coil had the wrong polarity once. We’re checking that. Also carefully checking for any obstructions, but it’s hard to see inside the DEE, so we’ll probably have to use a borescope.
August 21st, 2018:
- 3:08PM: The bad news is that even after fixing the major problem yesterday, the beam is behaving exactly the same; namely dropping off sharply at a particular radius. The “good” news is that we’ve tested this at multiple magnetic field settings and energies, and it’s always exactly the same radius. That makes if very unlikely that it’s anything subtle, like an instability. It’s probably something blocking the beam that was missed when we thought we found the problem yesterday. Our decision is to open up again and do a thorough inspection and clearance measurement. We will also thoroughly investigate the motion and positions of all three extraction elements.
- 9:16AM: After we closed it up, the conductor to the coax sprung an (unrelated) water leak. It’s fixed. Vacuum is good. We’re closing the vault now and will attempt to extract.
August 20th, 2018:
- 2:52PM: Good news! Randy and Michael opened up and found a (hopefully the) smoking gun. A linkage to the motion system for the ESD had come disconnected and was protruding into the beam path. It would explain everything we saw, and in fact it was sheer luck we were able to extract beam at 25 MeV. It’s been fixed and we’re pumping down. We should be at vacuum by tomorrow morning, when we’ll attempt to extract again. Stay tuned!
- 11:10AM: Unfortunately, we have had no luck extracting at 67.5 MeV. Using the probe, we’ve determined that the beam goes away about 1.5″ before the extraction radius. We had an idea that one of the old probes might have been obstructing the beam, so we went in and looked over the weekend, but that wasn’t the case. We are now attempting to extract the beam at a smaller radius, which will probably be an energy of about 61 MeV, but we’re never run with exactly that configuration, so it will take quite a bit of tuning. If that fails, we will make an access and remove the ESD, so we can fully inspect the machine to see of there’s some sort of occlusion we’ve missed. Contacting experts at TAMU and the LBNL to see if they have any ideas.
August 17th, 2018:
- 10:08AM: We continue to be unable to extract beam at 67.5 MeV. Trying to understand the problem.
August 16th, 2018:
- 11:24AM: Beam quickly re-established at 25 MeV. Going to tune at this energy until noon, then attempt 67 MeV again.
- 11:15AM: Back together and dried out. Turning back on…
- 10:14AM: The water problem recurred. We traced it to a blocked return line that prevented water flow, causing it to boil. The line has been cleared and we’re putting things back together.
August 15th, 2018:
- 3:51PM: A blown hose has ended our tuning for the day. It’s been fixed and we’ve placed fans to dry things out. Should be OK by morning.
- 3:16PM: Had some initial problems with efficiency, but after some tweaking, Randy was able to get decent currents. Still at 25 MeV, but at least it’s making sense.
- 2:05PM: Beam has been extracted! It’s still at 25 MeV, but it’s a big step. They’re going to spend the rest of the day tuning on 25 to understand the exact location of the extraction elements, then use this information to calculate the positions for 67 MeV.
- 11:16AM: Still had problems extracting beam, so on Tim’s recommendation, we are lowering the energy of the cyclotron, which, for somewhat esoteric reasons, makes extraction tuning significantly easier. Once we find the right position at low energy, it will be easier to find the right position at full energy.
August 14th, 2018:
- 3:09PM: After several hours of unsuccessfully trying to extract beam, it was discovered that there was a partial short between the coax lead and the mechanical linkage for the ESD. This was bleeding off current, making it impossible to extract beam. This has been repaired and they are returning to tuning.
- 11:31AM: Tim and Randy continue to try to tune for extraction. Having trouble finding the right settings for the ESD and coax. It might have something to do with alignment changes.
- 8:53AM: Once more unto the breach… Everything remained stable overnight. Randy is tuning and trying to extract.
August 13th, 2018:
- 5:00PM A/C Fixed. Verified that all interlocks make and all power supplies come on and stay on. Beam tomorrow morning for sure!
- 2:26PM Facilities located a motor and is installing it. Fingers crossed.
- 1:08PM Facilities determined the blower motor is bad. Looking for a spare in their warehouse. Worst case scenario is to rush order one for tomorrow.
- 11:48AM Facilities is here looking at the A/C.
- 11:19AM Our techs can’t fix the A/C motor, so we’ve called Facilities. We’re doing everything we can to light a fire under them.
- 10:45AM The trim supply was tripping on a thermal trip. The root cause is that the blower motor for the A/C in the pit is not working. This is yet another thing that has nothing to do with the shutdown. Hans and Michael are working on it. Can’t seem to catch a break.
- 8:58AM coax now on backup supply and seems to be working, but one of the trim supplies has failed. Hans is looking at it. It looks like an interlock relay. We’ll try to fix it, but can bypass it in the short term if need be.
August 10th, 2018:
- 3:01PM: The problem appears to be a faulty safety interlock in the control circuitry for the “coax” supply. This is unrelated to anything we did over the shutdown. Several attempts to diagnose and fix it failed, so the plan is to reroute the bus to the
- spare supply, so we will probably try again for beam on Monday morning.
- 12:46PM: The “coaxial” extraction element shorted every time Randy tried to adjust the position. Tim is investigating. Stay tuned.
- 11:53AM: Beam accelerating in cyclotron to outer radius. Attempting to extract.
- 11:07AM: Bringing up ion source. Arc established. Beginning to tune for acceleration.
- 10:46AM: Everything is turned on, including the electrostatic deflector (ESD) at full voltage. That’s is what prevented us from running before. Occasional sparks on the RF and ESD, which is normal after being exposed to air. Waiting for them to bake out a bit, then we’ll turn on the ion source and attempt to establish beam.
August 9th, 2018: Lots of minor “punch list” issues bringing the cyclotron back up, primarily involving some of the motors that control the mechanical elements of system. All fixed now. We believe everything is a go for bringing up beam first thing tomorrow.
August 7th, 2018: They are bringing of the RF today. It’s coming up smoothly, but it takes a bit of fine tuning on the shorting plate any time major work has been done to get the frequency right. In the mean time, after stalling a bit for outgassing, the vacuum is now down to 8e-6 torr, which I’m told is better than it’s been in about eight years, so all the hard work is paying off. Our hope is to try to establish beam starting tomorrow.
August 3rd, 2018: The vacuum is now 1.6e-5 Torr and dropping, which is historically good enough to run. We have brought up the magnet and the trim coils. The conductivity of the cooling water is a bit marginal after sitting for so long, so we are going to flush it over the weekend and changed the resin tank on Monday. Our goal is to turn on the RF system on Tuesday and attempt to establish beam.
August 1st, 2018: We believe the vacuum is good enough to turn the cyclotron on, and we’re working through lots of little problems associated with it being off so long, as well as some changes during the chilled water system during that time. Water flow has now been established, and the main magnet it turned on. We are now bringing up the trim coils.
July 27th, 2018: Vacuum 2.2e-5 Torr and falling. Fingers crossed.
July 25th, 2018: Additional leak found and fixed in extraction element. Roughing down again.
July 24th, 2018: Vacuum checking extraction element. So far so good. Hope to close up today or tomorrow and pump again.
July 23rd, 2018: Leak found in one of the extraction elements (the “iron”). Opened the cyclotron and it appears to be leaking solder joints. These were leak checked before, so the working assumption is that long exposure to air degraded them. Brian TIG-welded them (NB: TIG welding didn’t exist when the cyclotron was built).
July 20th, 2018: Pumping down, at 2e-5. Appears to be outgassing
Week of July 16th: Some small leaks found and repaired.
July 13th, 2018: On diffusion pumps. Vacuum at 5e-5
July 12th, 2018: Roughing down. Rapidly fell to 50 mTorr
July 10th, 2018: Systematic problem found with T-joints. Repaired with a combination of tightening and replacing o-ring segments.
July 6th, 2018: All flanges replaced. First attempt to rough down.

Summary of Refurbishment Project

Project Summary, including status, as of May 9th, 2018.

Pictures:

Here are some pictures from the project. The bulk of the work was carried out by Randy Kemmler, Brian Devine, Michael Backfish, Tim Essert, and Hans Berns.