Generates set.json files from Scryfall

I should have done this in the first place instead of figuring out how
to use MTGJSON files. Scryfall structures its data a bit differently and
notably includes a list of keywords for each card.

Having these set.json files enables easier analysis of the keyword
content of each set.
This commit is contained in:
Dan Niemitalo 2020-11-29 12:14:26 -06:00
parent 40f12289a2
commit 9dd4186e1c
2 changed files with 159 additions and 0 deletions

40
scryfall_requests.py Normal file
View file

@ -0,0 +1,40 @@
import requests
import json
import time
# Requests card objects from each set from Scryfall.com
# Takes a few seconds and creates a ~1 MB json file per set
# Reads set codes on separate lines in set_codes.txt
def request_set(code):
time.sleep(0.1) # Scryfall asks for 50-100 ms delay between requests
# Scryfall query is "set:code in:booster"
url = f'https://api.scryfall.com/cards/search?q=set%3A{code}+in%3Dbooster'
print(f'Requesting {code}')
response = requests.get(url)
d = response.json()
cards = []
cards += d['data']
while d['has_more']:
time.sleep(0.1)
print(f'Requesting {code} next page')
response = requests.get(d['next_page'])
d = response.json()
cards += d['data']
output_cards = json.dumps(cards)
if code == 'CON':
code = 'CON_'
f = open(f'scryfall/sets/{code}.json', 'w+')
f.write(output_cards)
f.close()
sets = []
set_file = open('set_codes.txt')
for line in set_file:
sets += [line.strip()]
set_file.close()
for code in sets:
request_set(code)

119
set_codes.csv Normal file
View file

@ -0,0 +1,119 @@
2ED
3ED
4ED
5ED
6ED
7ED
8ED
9ED
10E
M10
M11
M12
M13
M14
M15
ORI
M19
M20
M21
POR
P02
PTK
S99
ARN
ATQ
LEG
DRK
FEM
HML
ICE
ALL
MIR
VIS
WTH
TMP
STH
EXO
USG
ULG
UDS
MMQ
NEM
PCY
INV
PLS
APC
ODY
TOR
JUD
ONS
LGN
SCG
MRD
DST
5DN
CHK
BOK
SOK
RAV
GPT
DIS
CSP
TSP
PLC
FUT
LRW
MOR
SHM
EVE
ALA
CON_
ARB
ZEN
WWK
ROE
SOM
MBS
NPH
ISD
DKA
AVR
RTR
GTC
DGM
THS
BNG
JOU
KTK
FRF
DTK
BFZ
OGW
SOI
EMN
KLD
AER
AKH
HOU
XLN
RIX
DOM
GRN
RNA
WAR
ELD
THB
IKO
ZNR
MMA
MM2
EMA
MM3
IMA
A25
UMA
2XM
MH1
CMR
JMP
1 2ED
2 3ED
3 4ED
4 5ED
5 6ED
6 7ED
7 8ED
8 9ED
9 10E
10 M10
11 M11
12 M12
13 M13
14 M14
15 M15
16 ORI
17 M19
18 M20
19 M21
20 POR
21 P02
22 PTK
23 S99
24 ARN
25 ATQ
26 LEG
27 DRK
28 FEM
29 HML
30 ICE
31 ALL
32 MIR
33 VIS
34 WTH
35 TMP
36 STH
37 EXO
38 USG
39 ULG
40 UDS
41 MMQ
42 NEM
43 PCY
44 INV
45 PLS
46 APC
47 ODY
48 TOR
49 JUD
50 ONS
51 LGN
52 SCG
53 MRD
54 DST
55 5DN
56 CHK
57 BOK
58 SOK
59 RAV
60 GPT
61 DIS
62 CSP
63 TSP
64 PLC
65 FUT
66 LRW
67 MOR
68 SHM
69 EVE
70 ALA
71 CON_
72 ARB
73 ZEN
74 WWK
75 ROE
76 SOM
77 MBS
78 NPH
79 ISD
80 DKA
81 AVR
82 RTR
83 GTC
84 DGM
85 THS
86 BNG
87 JOU
88 KTK
89 FRF
90 DTK
91 BFZ
92 OGW
93 SOI
94 EMN
95 KLD
96 AER
97 AKH
98 HOU
99 XLN
100 RIX
101 DOM
102 GRN
103 RNA
104 WAR
105 ELD
106 THB
107 IKO
108 ZNR
109 MMA
110 MM2
111 EMA
112 MM3
113 IMA
114 A25
115 UMA
116 2XM
117 MH1
118 CMR
119 JMP