2024-09-09 00:00:39.437 [kb-webserver-75] INFO cn.kibibyte.shell.ApiInvoke - 收到游戏客户端消息[117.154.72.222][12976]:{"fn":"getchallenge","args":{"challenge":{"where":"data","cond":["challenge"],"value":{"vedio":1,"degree":2}},"type":"updateUI"},"build":1000,"base":{"appid":678,"agent":20,"channel":2,"os":1,"mode":"release","fsRatio":1.05,"ratio":0.4494195688225539,"ratio2":0.47089487402258906,"statusBar":110,"installid":"8529f991ac0e44acabae97c208cc0d26","language":"zh_CN_#Hans","total_disk":"469079","phone_name":"HUAWEI","cuntrycode":"CN","sys_version":"31","package_name":"com.july.qifujiaApp","total_memory":"7364","model":"FIN-AL60a","sdkver":1020,"ymid":"e4b5aab2363734fa53edef987801d9bcod","versionCode":104,"versionName":"2.0.4","accessToken":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIxIiwianRpIjoiZGZkYTM3M2FlZTJmNDE1MThiODJmZDkxN2U2NGU2OGU3ZTJlODc0NjA5NTQ3ZDM1ZjA4ZjY5ZWUzZmYzYWNhNzk1NDkzMDI4ZGYzODU0NWUiLCJpYXQiOjE3MjU0NDk2ODUuMTI0NDY2LCJuYmYiOjE3MjU0NDk2ODUuMTI0NDY4LCJleHAiOjE3NTY5ODU2ODUuMTE3NjcyLCJzdWIiOiI3MjcwOCIsInNjb3BlcyI6W119.ZfPJ9ps-99P3NKQKu9OF9_9eW8ihI1N6EFYyg0H5R06117wj4t07M5ZZ5Q2caMrt-wUpNPBrjNNhaaFo2eQsonB415JQLndWV7Nk3ezDAr04SSqOuwOh9DUnHB36m7D2DLIxxmclSZ0SjFsrJEnlh2DwC539wv2eZrRxnbmUiCgR6AXbT_NFabejZj2e6F3qDeTRcobhLTVSHN6fkJOFWIIL42nAEMKjcWgEtuQYpT1OCXY3BI0JbgmYZ1AkhYhSjXsskLpI877TR-Spj5TZOKkr3Du3n5S5Wmg45mfUoDxt4U9LT0UEQFox2WEEbnc2oom3W7bq9UkRCtpmTdQfbw","tokenType":"Bearer","uid":"72708"},"no":1055,"src":"tiaozhanshai","gate":{"remoteaddress":"117.154.72.222","uuid":"74119914-7f06-47a5-8d99-cdd315bdae4f","serialId":135962,"serverid":1001}} def read_large_json(file_path, batch_size=1000): encodings = 'utf-8' with open(file_path, 'r', encoding=encodings) as f: objects = ijson.items(f, 'item') chunk = [] for obj in objects: chunk.append(obj) if len(chunk) >= batch_size: yield pd.DataFrame(chunk) chunk = [] if chunk: yield pd.DataFrame(chunk) # 使用示例 for df_chunk in read_large_json('./project-2024-09-09-0.log'): print(df_chunk.shape)报错ijson.common.IncompleteJSONError: parse error: trailing garbage 2024-09-09 00:00:14.665 [kb-webserve (right here) ------^

### ijson.common.IncompleteJSONError 的解决方案 在使用 `ijson` 读取大 JSON 文件时，如果遇到 `IncompleteJSONError` 错误（例如 `trailing garbage`），这通常表明 JSON 文件的格式存在问题或文件内容不完整。以下是一些可能的原因及解决方案[^1]： #### 1. 检查 JSON 文件的格式 确保 JSON 文件的内容是有效的。可以使用在线工具（如 [JSONLint](https://jsonlint.com/)）验证 JSON 文件的格式是否正确。如果文件中存在多余的字符、未闭合的括号或其他语法错误，则会导致解析失败并抛出 `IncompleteJSONError`。 #### 2. 处理多段 JSON 数据 某些情况下，JSON 文件可能包含多个独立的 JSON 对象，而不是一个完整的 JSON 数组。这种情况下，`ijson` 会将额外的数据视为“尾随垃圾”（trailing garbage）。可以通过逐行读取文件并解析每一行中的 JSON 对象来解决此问题[^2]： ```python import json json_path = '../large_data/user.json' with open(json_path, "r") as f: for line in f: try: data = json.loads(line.strip()) print(data) except json.JSONDecodeError as e: print(f"JSON 解析错误: {e}") ``` #### 3. 使用自定义解析器忽略多余数据 如果 JSON 文件确实包含多余的尾随数据，可以尝试通过自定义解析逻辑跳过这些部分。例如，捕获 `IncompleteJSONError` 并继续处理剩余的有效数据： ```python import ijson json_path = '../large_data/user.json' with open(json_path, "r") as f: try: objects = ijson.items(f, 'item') for obj in objects: print(obj) except ijson.common.IncompleteJSONError as e: print(f"JSON 文件解析错误: {e}，可能是由于尾随垃圾导致") ``` #### 4. 分块读取大文件 对于超大 JSON 文件，建议采用分块读取的方式，以避免一次性加载整个文件到内存中。可以结合 `ijson` 和生成器实现流式读取： ```python import ijson json_path = '../large_data/user.json' chunk_size = 1024 * 1024 # 每次读取1MB def read_in_chunks(file_object, chunk_size): while True: data = file_object.read(chunk_size) if not data: break yield data with open(json_path, "r") as f: try: for chunk in read_in_chunks(f, chunk_size): objects = ijson.items(chunk, 'item') for obj in objects: print(obj) except ijson.common.IncompleteJSONError as e: print(f"JSON 文件解析错误: {e}") ``` #### 5. 验证文件完整性 如果 JSON 文件是从网络下载或传输过程中生成的，可能存在文件损坏或不完整的情况。建议重新检查文件来源，确保文件已完整下载或生成。 --- ### 示例代码：处理多段 JSON 数据 以下是一个完整的示例，展示如何处理包含多段 JSON 数据的大文件： ```python import json json_path = '../large_data/user.json' with open(json_path, "r") as f: for line in f: try: data = json.loads(line.strip()) print(data) except json.JSONDecodeError as e: print(f"JSON 解析错误: {e}") ``` --- ###