Magic was an Allied cryptanalysis project during World War II. It involved the United States Army's Signals Intelligence Service (SIS) and the United States Navy's Communication Special Unit.
Magic was set up to combine the US government's cryptologic capabilities in one organization dubbed the Research Bureau. Intelligence officers from the Army and Navy (and later civilian experts and technicians) were all under one roof. Although they worked on a series of codes and cyphers, their most important successes involved RED, BLUE, and PURPLE.
In 1923, a US Navy officer acquired a stolen copy of the Secret Operating Code codebook used by the Imperial Japanese Navy during World War I. Photographs of the codebook were given to the cryptanalysts at the Research Desk and the processed code was kept in red-colored folders (to indicate its Top Secret classification). This code was called "RED".[citation needed]
In 1930, the Japanese government created a more complex code that was codenamed BLUE, although RED was still being used for low-level communications. It was quickly broken by the Research Desk no later than 1932. US Military Intelligence COMINT listening stations began monitoring command-to-fleet, ship-to-ship, and land-based communications.[citation needed]
|
Main article: Purple code |
After Japan's ally Germany declared war in the fall of 1939, the German government began sending technical assistance to upgrade their communications and cryptography capabilities. One part was to send them modified Enigma machines to secure Japan's high-level communications with Germany. The new code, codenamed PURPLE (from the color obtained by mixing red and blue), was baffling.
PURPLE, like Enigma, began its communications with the same line of code but then became an unfathomable jumble. Codebreakers tried to break PURPLE communiques by hand but found they could not. Then the codebreakers realized that it was not a manual additive or substitution code like RED and BLUE, but a machine-generated code similar to Germany's Enigma cipher. Decoding was slow and much of the traffic was still hard to break. By the time the traffic was decoded and translated, the contents were often out of date.
A reverse-engineered machine created in 1939 by a team of technicians led by William Friedman and Frank Rowlett could decrypt some of the PURPLE code by replicating some of the settings of the Japanese Enigma machines. This accelerated decoding and the addition of more translators on staff in 1942 made it easier and quicker to decipher the traffic intercepted.[citation needed]
The Japanese Foreign Office used a cipher machine to encrypt its diplomatic messages. The machine was called "PURPLE" by U.S. cryptographers. A message was typed into the machine, which enciphered and sent it to an identical machine. The receiving machine could decipher the message only if set to the correct settings, or keys. American cryptographers built a machine that could decrypt these messages.
The PURPLE machine itself was first used by Japan in 1940. U.S. and British cryptographers had broken some PURPLE traffic well before the attack on Pearl Harbor. However, the PURPLE machines were used only by the Foreign Office to carry diplomatic traffic to its embassies. The Japanese Navy used a completely different crypto-system, known as JN-25.
U.S. analysts discovered no hint in PURPLE of the impending Japanese attack on Pearl Harbor. Nor could they, as the Japanese were very careful not to discuss their plan in Foreign Office communications. No detailed information about the planned attack was even available to the Japanese Foreign Office, as that agency was regarded by the military, particularly its more nationalist members, as insufficiently "reliable".
U.S. access to private Japanese diplomatic communications (even the most secret ones) was less useful than it might otherwise have been because policy in prewar Japan was controlled largely by military groups like the Imperial Way Faction, and not by the Foreign Office. The Foreign Office itself deliberately withheld from its embassies and consulates much of the information it did have, so the ability to read PURPLE messages was less than definitive regarding Japanese tactical or strategic military intentions.
U.S. cryptographers (see Station HYPO) had decrypted and translated the 14-part Japanese diplomatic message breaking off ongoing negotiations with the U.S. at 1 p.m. Washington time on 7 December 1941, even before the Japanese Embassy in Washington could do so. As a result of the deciphering and typing difficulties at the embassy, the note was delivered late to American Secretary of State Cordell Hull. When the two Japanese diplomats finally delivered the note, Hull had to pretend to be reading it for the first time, even though he already knew about the attack on Pearl Harbor.[1]
Throughout the war, the Allies routinely read both German and Japanese cryptography. The Japanese Ambassador to Germany, General Hiroshi Ōshima, often sent priceless German military information to Tokyo. This information was routinely intercepted and read by Roosevelt, Churchill and Eisenhower.[2] According to Lowman, "The Japanese considered the PURPLE system absolutely unbreakable… Most went to their graves refusing to believe the [cipher] had been broken by analytic means… They believed someone had betrayed their system."[3]
Even so, the diplomatic information was of limited value to the U.S. because of its manner and its description. "Magic" was distributed in such a way that many policy-makers who had need of the information in it knew nothing of it, and those to whom it actually was distributed (at least before Pearl Harbor) saw each message only briefly, as the courier stood by to take it back, and in isolation from other messages (no copies or notes being permitted). Before Pearl Harbor, they saw only those decrypts thought "important enough" by the distributing Army or Navy officers.
Nonetheless, being able to read PURPLE messages gave the Allies a great advantage in the war. For instance, the Japanese ambassador to Germany, Baron Hiroshi Ōshima, produced long reports for Tokyo which were enciphered on the PURPLE machine. They included reports on personal discussions with Adolf Hitler and a report on a tour of the invasion defenses in Northern France (including the D-Day invasion beaches). General Marshall said that Ōshima was "our main basis of... information regarding Hitler's intentions in Europe".
During the 1944 election, Thomas Dewey threatened to make Pearl Harbor a campaign issue,[4] until General Marshall sent him a personal letter which said, in part:
Dewey promised not to raise the issue, and kept his word.
The break into the PURPLE system, and into Japanese messages generally, was the subject of acrimonious hearings in Congress post-World War II in connection with an attempt to decide who, if anyone, had allowed the disaster at Pearl Harbor to happen and who therefore should be blamed. During those hearings the Japanese learned, for the first time, the PURPLE cypher system had been broken. They had been continuing to use it, even after the War, with the encouragement of the American Occupation Government.
Much confusion over who in Washington or Hawaii knew what and when, especially as "we were decrypting their messages," has led some to conclude "someone in Washington" knew about the Pearl Harbor attack before it happened, and, since Pearl Harbor was not expecting to be attacked, the "failure to warn Hawaii one was coming must have been deliberate, since it could hardly have been mere oversight".[citation needed] However, PURPLE was a diplomatic, not a military code. Thus, only inferences could be drawn from PURPLE as to specific Japanese military actions.
When PURPLE was broken by the U.S. Army's Signals Intelligence Service (SIS),[5] several problems arose for the Americans: who would get the decrypts, which decrypts, how often, under what circumstances, and crucially (given interservice rivalries) who would do the delivering. Both the U.S. Navy and Army were insistent they alone handle all decrypted traffic delivery, especially to highly placed policy makers in the U.S. Eventually, after much to-ing and fro-ing, a compromise was reached: the Army would be responsible for the decrypts on one day, and the Navy the next.
The distribution list eventually included some—but not all—military intelligence leaders in Washington and elsewhere, and some—but, again, not all—civilian policy leaders in Washington. The eventual routine for distribution included the following steps:
There were several prior steps needed before any decrypt was ready for distribution:
In the period before the attack on Pearl Harbor, the material was handled awkwardly and inefficiently, and was distributed even more awkwardly. Nevertheless, the extraordinary experience of reading a foreign government's most closely held communications, sometimes even before the intended recipient, was astonishing. It was so astonishing, someone (possibly President Roosevelt) called it magic. The name stuck.
One aspect of Magic remains controversial to this day—the amount of involvement the intercepts played in the issuing of United States Executive Order 9066 on February 19, 1942, and subsequent Executive Order 9102 on March 18, which led to the creation of the Wartime Relocation Authority (WRA). This is often confused with the issue of internment, which was actually handled by the Justice Department's Immigration and Naturalization Service (INS) and affected all citizens of countries at war with the United States in any location.
Internment of "enemy aliens" by the U.S. government began two months prior to Executive Order 9066 on December 8, 1941, immediately after the attack at Pearl Harbor and included Germans and Italians, and not just the Japanese living on the U.S. West Coast.[6]
David Lowman in his book MAGIC: the Untold Story[7] reports that the primary justification for the Japanese-American relocations and internments was to protect against espionage and sabotage, because Magic could not be mentioned during the war. Those defending the decision to evacuate and relocate when seen in context, notably blogger and investigative reporter Michelle Malkin, point to Magic intercepts as partial justification for EO 9066. Malkin cites 1984 testimony of the Undersecretary with the most Magic knowledge, who stated[8] that Magic "was a very important factor" in their considerations.
Extensive additional primary source documents are cited in Malkin's book In Defense of Internment[9] to argue that Magic intercepts discuss the development of a spy ring among Japanese Americans by the Japanese consulates, provide the type of espionage data being sent to Japan, and much more which raised a suspicion that many thousands in the Japanese American community were an espionage risk, including members of Kibei, Issei and Nisei.
In 1988, Congress passed and President Ronald Reagan signed legislation that apologized for the internment on behalf of the U.S. government. The legislation said that government actions were based on "race prejudice, war hysteria, and a failure of political leadership".[10] The hearings that produced this decision did not take into account the Magic intercepts.[8]
The following is the actual text of several Magic intercepts translated into English before and during the war and declassified and made public in 1978 by the U.S. government (The Magic Background of Pearl Harbor:, Government Printing Office, 8 volumes)
Intercept dated January 30, 1941 and noted as translated 2-7-41 Numbered #44
FROM: Tokyo (Matsuoka)
TO: Washington (Koshi)
(In two parts – complete). (Foreign Office secret).
(1) Establish an intelligence organ in the Embassy which will maintain liaison with private and semi-official intelligence organs (see my message to Washington #591 and #732 from New York to Tokyo, both of last year's series). With regard to this, we are holding discussions with the various circles involved at the present time.
(2) The focal point of our investigations shall be the determination of the total strength of the U.S. Our investigations shall be divided into three general classifications: political, economic, and military, and definite course of action shall be mapped out.
(3) Make a survey of all persons or organizations which either openly or secretly oppose participation in the war.
(4) Make investigations of all antisemitism, communism, movements of Negroes, and labor movements.
(5) Utilization of U.S. citizens of foreign extraction (other than Japanese), aliens (other than Japanese), communists, Negroes, labor union members, and anti-Semites, in carrying out the investigations described in the preceding paragraph would undoubtedly bear the best results. These men, moreover, should have access to governmental establishments, (laboratories?), governmental organizations of various characters, factories, and transportation facilities.
(6) Utilization of our "Second Generations" and our resident nationals. (In view of the fact that if there is any slip in this phase, our people in the U.S. will be subjected to considerable persecution, and the utmost caution must be exercised).
(7) In the event of U.S. participation in the war, our intelligence set-up will be moved to Mexico, making that country the nerve center of our intelligence net. Therefore, will you bear this in mind and in anticipation of such an eventuality, set up facilities for a U.S.-Mexico international intelligence route. This net which will cover Brazil, Argentina, Chile, and Peru will also be centered in Mexico.
(8) We shall cooperate with the German and Italian intelligence organs in the U.S. This phase has been discussed with the Germans and Italians in Tokyo, and it has been approved.
Please get the details from Secretary Terasaki upon his assuming his duties there.
Please send copies to those offices which were on the distribution list of No. 43.
Throughout the rest of 1941, some of the messages between Tokyo and its embassies and consulates continued to be intercepted.
In response to the ordered shift from propaganda efforts to espionage collection, the Japanese consulates throughout the western hemisphere reported their information normally through the use of diplomatic channels, but when time-sensitive through the use of PURPLE encoded messages. This provided vital clues to their progress directly to the U.S. President and his top advisers.
Intercepts in May 1941 from the consulates in Los Angeles and Seattle report that the Japanese were having success in obtaining information and cooperation from "second generation" Japanese Americans and others.
Intercept dated May 9, 1941 and translated 5-19-41 Numbered #067
FROM: Los Angeles (Nakauchi)
TO: Tokyo (Gaimudaijin)
(In 2 parts – complete). Strictly Secret.
Re your message # 180 to Washington.
We are doing everything in our power to establish outside contacts in connection with our efforts to gather intelligence material. In this regard, we have decided to make use of white persons and Negroes, through Japanese persons whom we cannot trust completely. (It not only would be very difficult to hire U.S. (military?) experts for this work at present time, but the expenses would be exceedingly high.) We shall, furthermore, maintain close connections with the Japanese Association, the Chamber of Commerce, and the newspapers.
With regard to airplane manufacturing plants and other military establishments in other parts, we plan to establish very close relations with various organizations and in strict secrecy have them keep these military establishments under close surveillance. Through such means, we hope to be able to obtain accurate and detailed intelligence reports. We have already established contacts with absolutely reliable Japanese in the San Pedro and San Diego area, who will keep a close watch on all shipments of airplanes and other war materials, and report the amounts and destinations of such shipments. The same steps have been taken with regards to traffic across the U.S.-Mexico border.
We shall maintain connection with our second generations who are at present in the (U.S.) Army, to keep us informed of various developments in the Army. We also have connections with our second generations working in airplane plants for intelligence purposes.
With regard to the Navy, we are cooperating with our Naval Attache's office, and are submitting reports as accurately and speedily as possible.
We are having Nakazawa investigate and summarize information gathered through first hand and newspaper reports, with regard to military movements, labor disputes, communistic activities and other similar matters. With regard to anti-Jewish movements, we are having investigations made by both prominent Americans and Japanese who are connected with the movie industry which is centered in this area. We have already established connections with very influential Negroes to keep us informed with regard to the Negro movement.
Intercept dated May 11, 1941 and translated 6-9-41 Numbered # 45
FROM: Seattle (Sato)
TO: Tokyo
(3 parts – complete)
Re your # 180 to Washington
1. Political Contacts We are collecting intelligences revolving around political questions, and also the questions of American participation in the war which has to do with the whole country and this local area.
2. Economic Contacts We are using foreign company employees, as well as employees in our own companies here, for the collection of intelligence having to do with economics along the lines of the construction of ships, the number of airplanes produced and their various types, the production of copper, zinc and aluminum, the yield of tin for cans, and lumber. We are now exerting our best efforts toward the acquisition of such intelligences through competent Americans. From an American, whom we contacted recently, we have received a private report on machinists of German origin who are Communists and members of the labor organizations in the Bremerton Naval Yard and Boeing airplane factory. Second generation Japanese ----- ----- ----- [three words missing].
3. Military Contacts We are securing intelligences concerning the concentration of warships within the Bremerton Naval Yard, information with regard to mercantile shipping and airplane manufacturing, movements of military forces, as well as that which concerns troop maneuvers. With this as a basis, men are sent out into the field who will contact Lt. Comdr. OKADA, and such intelligences will be wired to you in accordance with past practice. KANEKO is in charge of this. Recently we have on two occasions made investigations on the spot of various military establishments and concentration points in various areas. For the future we have made arrangements to collect intelligences from second generation Japanese draftees on matters dealing with the troops, as well as troop speech and behavior. ----- ---- -----. [three words missing]
4. Contacts with Labor Unions The local labor unions A.F. of L. and C.I.O. have considerable influence. The (Socialist?) Party maintains an office here (its political sphere of influence extends over twelve zones.) The C.I.O., especially, has been very active here. We have had a first generation Japanese, who is a member of the labor movement and a committee chairman, contact the organizer, and we have received a report, though it is but a resume, on the use of American members of the (Socialist ?) Party. ------ OKAMARU is in charge of this.
5. In order to contact Americans of foreign extraction and foreigners, in addition to third parties, for the collection of intelligences with regard to anti-participation organizations and the anti-Jewish movement, we are making use of a second generation Japanese lawyer.
This intelligence ---- ----- -----.
These intercepts plus other reports from the FBI and the Office of Naval Intelligence counter-espionage efforts, the TACHIBANA espionage case during summer 1941, FBI efforts against Japanese yakuza throughout the 1930s along the West Coast (the TOKOYO and TOYO CLUBs) were all available only to the most senior leaders in the Roosevelt cabinet. Even J. Edgar Hoover, Director of the FBI, was not privy to the existence of Magic intelligence.[citation needed]
Those who consider that Executive Order 9066 regarding Japanese American internment was not based on Magic intercepts, argue[citation needed]:
One theory is that Stimson wanted DeWitt to provide justifications that could be made public, because the Magic intercepts could not be made public.
The issue was inflamed due to the release of Malkin's 2004 book, In Defense of Internment, in which the Magic intercepts play a major role in the defense of her thesis.
PURPLE was an enticing, but quite tactically limited, window into Japanese planning and policy because of the peculiar nature of Japanese policy making prior to the War (see above). Early on, a better tactical window was the Japanese Fleet Code (an encoded cypher), called JN-25 by U.S. Navy cryptanalysts. Breaking into the version in use in the months after December 7, 1941 provided enough information to lead to U.S. naval victories in the battles of the Coral Sea and Midway, stopping the initial Japanese advances to the south and eliminating the bulk of Japanese naval air power.
Later, broken JN-25 traffic provided the schedule and routing of the plane Admiral Isoroku Yamamoto would be flying in during an inspection tour in the southwest Pacific, giving USAAF pilots a chance to ambush the officer who had conceived the Pearl Harbor attack. Later still, access to Japanese Army messages from decrypts of Army communications traffic assisted in planning the island hopping campaign to the Philippines and beyond.
Another source of information was the Japanese Military Attaché code (known as JMA to the Allies) introduced in 1941. This was a fractionating transposition system based on two-letter code groups which stood for common words and phrases. The groups were written in a square grid according to an irregular pattern and read off vertically, similar to a disrupted columnar transposition. Then the letters were superenciphered using a pre-arranged table of alphabets. This system was broken by John Tiltman at Bletchley Park in 1942.[11]
The 1992 book The Sword and the Shield: The Mitrokhin Archive and the Secret History of the KGB, by Christopher Andrew, based on the Mitrokhin Archive smuggled out of Russia in the early 1990s by a KGB archivist, contains information about wartime Soviet knowledge of Japanese enciphered transmissions. It claims that the Soviets independently broke into Japanese PURPLE traffic (as well as the Red predecessor machine).
It claims that decrypted PURPLE messages contributed to the decision by Stalin to move troops from Far Eastern Asia to the area around Moscow for the counterattack against Germany in December 1941, as the messages convinced the Soviet government that there would not be a Japanese attack.
Public notice had actually been served that Japanese cryptography was dangerously inadequate by the Chicago Tribune, which published a series of stories just after Midway, starting on 7 June 1942, which claimed (correctly) that victory was due in large part to the U.S. breaking into Japanese crypto systems (in this case, the JN-25 cypher, though which system(s) had been broken was not mentioned in the newspaper stories). The Tribune claimed the story was written by Stanley Johnston from his own knowledge (and Jane's).[12]
Ronald Lewin points out that the story repeats the layout and errors of a signal from Admiral Nimitz which Johnston saw while on the transport Barnett. Nimitz was reprimanded by Admiral King for sending the dispatch to Task Force commanders on a channel available to nearly all ships.[12] The Lexington's executive officer, Commander Morton T. Seligman was assigned to shore duty and retired early.
However, neither the Japanese nor anyone who might have told them seem to have noticed either the Tribune coverage, or the stories based on the Tribune account published in other U.S. papers. Nor did they notice announcements made on the floor of the United States Congress to the same effect. There were no changes in Japanese cryptography connected with those newspaper accounts or Congressional disclosures.
Alvin Kernan was an aviation ordnanceman on board the aircraft carriers Enterprise and the Hornet during the war. During that time, he was awarded the Navy Cross. In his book Crossing the Line, he states that when the carrier returned to Pearl Harbor to resupply before the Battle of Midway, the crew knew that the Japanese code had been broken and that U.S. naval forces were preparing to engage the Japanese fleet at Midway. He insists that he "…exactly remembers the occasion on which I was told, with full details about ships and dates…" despite the later insistence that the breaking of the code was kept secret.[13]
U.S. Navy Commander I.J. Galantin, who retired as an Admiral, refers several times to Magic in his 1988 book about his Pacific theater war patrols as captain of the U.S. submarine Halibut. However, Galantin refers to Magic as "Ultra" which was actually the name given to the breaking of the German code. Upon receiving one message from Pacific Fleet command, directing him off normal station to intercept Japanese vessels due to a Magic message, Galantin writes. "I had written my night orders carefully. I made no reference to Ultra and stressed only the need to be very alert for targets in this fruitful area". Galantin had previously mentioned in his book that all submarine captains were aware of "Ultra" (Magic).[14]
In addition, Army Chief of Staff George C. Marshall discovered early in the war that Magic documents were being widely read at the White House, and that "…at one time over 500 people were reading messages we had intercepted from the Japanese… Everyone seemed to be reading them" .[15]
Neal Stephenson's novel Cryptonomicon includes a fictionalized version of Magic, with the Japanese cryptosystem being named "Indigo" rather than "PURPLE".
James Bond is given the products of the fictionalized "MAGIC 44" decryption programme in You Only Live Twice as a bargaining chip when he is deployed to negotiate for intelligence concessions from Tiger Tanaka, head of Japanese intelligence.
The W.E.B. Griffin series The Corps is a fictionalized account of United States Navy and Marine Corps intelligence operations in the Pacific Theater during World War II. Many of the main characters in the novels, both fictional and historical, have access to and use intelligence from Magic.