Stereotactic Radiation Therapy and Ipilimumab in Treating Patients With Metastatic Melanoma

Inclusion Criteria: * Willing and able to give written informed consent * Eastern Cooperative Oncology Group (ECOG) performance status 0-2 * Histologic diagnosis of melanoma with metastatic disease to a visceral organ (lung, liver, brain, adrenal, nodal station outside the regional lymph drainage of the primary, vertebral bodies) * 1-3 sites of metastatic disease able to be targeted by SABR * White blood cells (WBC) \>= 2000/uL * Absolute neutrophil count (ANC) \>= 1000/uL * Platelets \>= 75 x 10\^3/uL * Hemoglobin \>= 9 g/dL (\>= 80 g/L; may be transfused) * Creatinine =\< 2.0 x upper limit of normal (ULN) * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 2.5 x ULN for patients without liver metastasis, =\< 5 times for liver metastases * Bilirubin =\< 2.0 x ULN, (except patients with Gilbert's syndrome, who must have a total bilirubin less than 3.0 mg/dL) * No active or chronic infection with human immunodeficiency virus (HIV), hepatitis B, or hepatitis C * Women of childbearing potential (WOCBP) must be using an adequate method of contraception to avoid pregnancy throughout the study and for up to 26 weeks after the last dose of investigational product, in such a manner that the risk of pregnancy is minimized * WOCBP include any female who has experienced menarche and who has not undergone successful surgical sterilization (hysterectomy, bilateral tubal ligation, or bilateral oophorectomy) or is not post-menopausal; post-menopause is defined as: * Amenorrhea \>= 12 consecutive months without another cause, or * For women with irregular menstrual periods and taking hormone replacement therapy (HRT), a documented serum follicle stimulating hormone (FSH) level \>= 35 mIU/mL * Women who are using oral contraceptives, other hormonal contraceptives (vaginal products, skin patches, or implanted or injectable products), or mechanical products such as an intrauterine device or barrier methods (diaphragm, condoms, spermicides) to prevent pregnancy, or are practicing abstinence or where their partner is sterile (eg, vasectomy) should be considered to be of childbearing potential * WOCBP must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within 72 hours before the start of ipilimumab * Men of fathering potential must be using an adequate method of contraception to avoid conception throughout the study (and for up to 26 weeks after the last dose of investigational product) in such a manner that the risk of pregnancy is minimized Exclusion Criteria: * Any other malignancy from which the patient has been disease-free for less than 3 years, with the exception of adequately treated and cured basal or squamous cell skin cancer, superficial bladder cancer or carcinoma in situ of the cervix * Autoimmune disease: patients with a history of inflammatory bowel disease, including ulcerative colitis and Crohn's disease, are excluded from this study, as are patients with a history of symptomatic disease (eg, rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, autoimmune vasculitis \[eg, Wegener's Granulomatosis\]); motor neuropathy considered of autoimmune origin (e.g. Guillain-Barre syndrome and Myasthenia Gravis) * Any underlying medical or psychiatric condition, which in the opinion of the investigator will make the administration of ipilimumab hazardous or obscure the interpretation of adverse events (AEs), such as a condition associated with frequent diarrhea * Any non-oncology vaccine therapy used for prevention of infectious diseases (for up to 1 month before or after any dose of ipilimumab) * A history of prior treatment with ipilimumab or prior cluster of differentiation (CD)137 agonist or cytotoxic T-lymphocyte antigen 4 (CTLA-4) inhibitor or agonist * A history of prior treatment with anti-programmed death (PD)-1 or anti-PD-L1 antibodies * Concomitant therapy with any of the following: interleukin (IL)-2, interferon, other non-study immunotherapy regimens, cytotoxic chemotherapy, other investigation therapies * Concomitant therapy with immune-suppressants or chronic use of systemic corticosteroids * Must be off prior systemic therapies for 2 weeks prior to enrollment; patients that have been previously treated with systemic therapy adjuvantly or for metastatic disease remain eligible as long as they continue to meet all other eligibility criteria (oligometastatic, no visceral metastasis \> 5 cm, eligible for SABR) * Prior radiation therapy that at the treating physician's discretion makes SABR unsafe * No evidence of pleural effusion or ascites * Congestive heart failure \> class II New York Heart Association (NYHA) or unstable angina * Cardiac ventricular arrhythmias requiring anti-arrhythmic therapy * Major surgery, open biopsy or significant traumatic injury within 2 weeks of first dose of study drug * A visceral metastasis greater than 5 cm * A visceral metastasis that due to its location cannot be safely treated with SABR * Women of childbearing potential (WOCBP), defined above who: * Are unwilling or unable to use an acceptable method of contraception to avoid pregnancy for their entire study period and for at least 8 weeks after cessation of study drug, or * Have a positive pregnancy test at baseline, or * Are pregnant or breastfeeding * Prisoners or subjects who are compulsorily detained (involuntarily incarcerated) for treatment of either a psychiatric or physical (eg, infectious) illness * Have a known immediate or delayed hypersensitivity reaction or idiosyncrasy to drugs chemically related to study drug, or excipients or to dimethyl sulfoxide (DMSO) * Persons of reproductive potential must agree to use an adequate method of contraception throughout treatment and for at least 8 weeks after ipilimumab is stopped * Sexually active WOCBP must use an effective method of birth control during the course of the study, in a manner such that risk of failure is minimized; before study enrollment, WOCBP must be advised of the importance of avoiding pregnancy during study participation and the potential risk factors for an unintentional pregnancy; all WOCBP MUST have a negative pregnancy test before first receiving ipilimumab; if the pregnancy test is positive, the patient must not receive ipilimumab and must not be enrolled in the study