如何实现分布式文件上传解决方案？

I have a file uploading site which is currently resting on a single server i.e using the same server for users to upload the files to and the same server for content delivery.

我有一个文件上传站点，该站点目前停留在单个服务器上，即使用相同的服务器供用户上传文件到同一服务器进行内容交付。

What I want to implement is a CDN (content delivery network). I would like to buy a server farm and somehow if i were to have a mechanism to have files spread out across the different servers, that would balance my load a whole lot better.

我想要实现的是CDN（内容分发网络）。我想买一个服务器场，不知怎的，如果我有一个机制让文件分散在不同的服务器上，这将平衡我的负载更好。

However, I have a few questions regarding this:

但是，我对此有几个问题：

Assuming my server farm consists of 10 servers for content delivery,

假设我的服务器场包含10台用于内容交付的服务器，

Since at the user end, the script to upload files will be one location only, i.e <form action=upload.php>, It has to reside on a single server, correct? How can I duplicate the script across multiple servers and direct the user's file upload data to the server with the least load?
因为在用户端，上传文件的脚本只是一个位置，即

，它必须驻留在一台服务器上，对吗？如何跨多个服务器复制脚本并将用户的文件上载数据导向负载最小的服务器？
How should I determine which files to be sent to which server? During the upload process, should I randomize all files to go to random servers? If the user sends 10 files should i send them to a random server? Is there a mechanism to send them to the server with the least load? Is there any other algorithm which can help determine which server the files need to be sent to?
我该如何确定将哪些文件发送到哪台服务器？在上传过程中，我应该随机将所有文件转到随机服务器吗？如果用户发送10个文件，我应该将它们发送到随机服务器吗？是否有机制将它们发送到负载最小的服务器？有没有其他算法可以帮助确定需要将文件发送到哪个服务器？
How will the files be sent from the upload server to the CDN? Using FTP? Wouldn't that introduce additional overhead and need for error checking capability to check for FTP connection break, and to check if file was transferred successfully etc.?
如何将文件从上传服务器发送到CDN？使用FTP？这不会引入额外的开销，需要错误检查功能来检查FTP连接中断，并检查文件是否已成功传输等？

1 个解决方案

#1

Assuming you're using an Apache server, there is a module called mod_proxy_balancer. It handles all of the load-balancing work behind the scenes. The user will never know the difference -- except when their downloads and uploads are 10 times faster.

假设您正在使用Apache服务器，那么有一个名为mod_proxy_balancer的模块。它处理幕后的所有负载平衡工作。用户永远不会知道差异 - 除非他们的下载和上传速度快10倍。

If you use this, you can have a complete copy on each server.
如果使用此功能，则可以在每台服务器上获得完整副本。
mod_proxy_balancer will handle this for you.
mod_proxy_balancer将为您处理此问题。
Each server can have its own sub-domain. You will have a database on your 'main' server, which matches up all of your download pages to the physical servers they are located on. Then a on-the-fly URL is passed based on some hash encryption algorithm, which prevents using a hard link to the download and increases your page hits. It could be a mix of personal and miscellaneous information, e.g., the users IP and the time of day. The download server then checks the hashes, and either accepts or denies the request.
每个服务器都可以拥有自己的子域。您将在“主”服务器上拥有一个数据库，该数据库将所有下载页面与它们所在的物理服务器相匹配。然后基于一些散列加密算法传递动态URL，这会阻止使用下载的硬链接并增加页面命中率。它可以是个人信息和杂项信息的混合，例如用户IP和一天中的时间。然后，下载服务器检查哈希值，并接受或拒绝请求。

If everything checks out, the download starts; your load is balanced; and the users don't have to worry about any of this behind the scenes stuff.

如果一切都结束，下载开始;你的负荷是平衡的;并且用户不必担心任何这些幕后的东西。

note: I have done Apache administration and web development. I have never managed a large CDN, so this is based on what I have seen in other sites and other knowledge. Anyone who has something to add here, or corrections to make, please do.

注意：我已经完成了Apache管理和Web开发。我从未管理过大型CDN，所以这是基于我在其他网站和其他知识中看到的内容。任何有兴趣在这里添加或者更正的人都可以。

Update

There are also companies that manage it for you. A simple Google search will get you a list.

也有公司为您管理它。一个简单的谷歌搜索将为您提供一个列表。

1 个解决方案

#1

Update

更多相关文章

随机推荐