这python数据如何去重？

Question

些了个脚本爬取了些数据。但是有很多重复的。要怎么去冲后，写入csv文件？下面是我写的代码。

with open('D:\\result3.csv','w') as f:
    writer = csv.writer(f)
    writer.writerow(['company_name','emails', 'website','url'])
    for t in a:
        company_name = t['company_name']
        email = t['emails']
        url = t['url']
        web = t['website']

        data = [(company_name,email, web, url)]
        writer.writerows(data)
f.close()

譬如说如果email已经存在了，就不写入csv文件，包括company_name,url,web，都不写入csv。
像这样要如何去重，且在不打乱源数据。

Answer 1

小数据可以这样写

with open('D:\result3.csv','w') as f:

    writer = csv.writer(f)
    writer.writerow(['company_name','emails', 'website','url'])
    
    email_dict = {}
    
    for t in a:
        company_name = t['company_name']
        email = t['emails']
        url = t['url']
        web = t['website']
        if email in email_dict:
             continue
        data = [(company_name,email, web, url)]
        writer.writerows(data)
        
        email_dict[email] = 1

Answer 2

对于数据量不大的情况可以使用字典来完成