项目介绍最近学习了多线程相关知识，通过一个小项目对所学知识梳理，做一个综合的运用。
项目演示该项目主要是使用HttpURLConection发起HTTP请求，再结合IO流和多线程对文件进行一个切分下载，最后合并。
项目目录结构
代码
项目入口类，需要传入下载地址，或者在控制台输入
public class Main {
public static void main(String[] args) {
//下载地址
String url = null;
if (args == null || args.length == 0) {
while (url == null) {
LogUtils.info("请输入下载地址");
Scanner scanner = new Scanner(System.in);
url = scanner.next();
}
}else {
url = args[0];
}
Downloader downloader = new Downloader();
downloader.download(url);
}
}
通过项目入口类我们可以发现，整个项目的细节都在Downloader这个类中，要想弄清楚Downloader类中的细节，我们先把系统工具好好看一看
HttpUtils，主要通过这个工具类获取HTTP请求对象，获取所下载文件的相关信息，如：文件大小、文件名字、分块下载等。
public class HttpUtils {
public static long getHttpFileContentLength(String url) throws IOException {
int contentLength;
HttpURLConnection httpURLConnection = null;
try {
httpURLConnection = getHttpURLConnection(url);
contentLength = httpURLConnection.getContentLength();
} finally {
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
return contentLength;
}
public static HttpURLConnection getHttpURLConnection(String url, long startPos, long endPos) throws IOException {
HttpURLConnection httpURLConnection = getHttpURLConnection(url);
LogUtils.info("下载的区间是：{}-{}",startPos,endPos);
if (endPos != 0) {
httpURLConnection.setRequestProperty("RANGE","bytes=" + startPos + "-" + endPos);
}else {
httpURLConnection.setRequestProperty("RANGE","bytes=" + startPos + "-");
}
return httpURLConnection;
}
public static HttpURLConnection getHttpURLConnection(String url) throws IOException {
URL httpUrl = new URL(url);
HttpURLConnection httpURLConnection = (HttpURLConnection) httpUrl.openConnection();
//向文件所在的服务器发送标识信息
httpURLConnection.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1");
return httpURLConnection;
}
public static String getHttpFileName(String url){
return url.substring(url.lastIndexOf("/") + 1);
}
}
FileUtils中的getFileContentLength方法，主要是用来判断该文件有没有重复下载
public class FileUtils {
public static long getFileContentLength(String path){
File file = new File(path);
return file.exists() && file.isFile() ? file.length() : 0;
}
}
LogUtils 自定义日志工具类，提供了统一的日志管理，方便阅读。
public class LogUtils {
public static void info(String msg,Object... args){
print(msg,"-info-",args);
}
public static void error(String msg,Object... args){
print(msg,"-error-",args);
}
private static void print(String msg,String level,Object... args){
if (args != null && args.length > 0) {
msg = String.format(msg.replace("{}","%s"),args);
}
String name = Thread.currentThread().getName();;
System.out.println(LocalTime.now().format(DateTimeFormatter.ofPattern("hh:mm:ss")) + " " +
name + level + msg);
}
}
Downloader实现细节
scheduledExecutorService线程池，是用来打印实时的下载信息，比如下载速度什么的。poolExecutor线程池，是用来进行分块下载的，将文件分为多个小块，多个线程并发下载。根据阿里巴巴代码规范手册，最好使用原生方法创建线程池，我这里演示了两种创建方法
public class Downloader {
private final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1);
private final ThreadPoolExecutor poolExecutor = new ThreadPoolExecutor(Constant.THREAD_NUM,Constant.THREAD_NUM,
0,TimeUnit.SECONDS,new ArrayBlockingQueue<>(Constant.THREAD_NUM));
private final CountDownLatch countDownLatch = new CountDownLatch(Constant.THREAD_NUM);
public void download(String url){
//获取文件名
String httpFileName = HttpUtils.getHttpFileName(url);
//文件下载路径
httpFileName = Constant.PATH + httpFileName;
//获取本地文件大小
long localFileLength = FileUtils.getFileContentLength(httpFileName);
HttpURLConnection httpURLConnection = null;
DownloadInfoThread downloadInfoThread = null;
//获取连接对象
try {
httpURLConnection = HttpUtils.getHttpURLConnection(url);
//获取下载文件的总大小
int contentLength = httpURLConnection.getContentLength();
//文件是否已下载过
if (localFileLength >= contentLength) {
LogUtils.info("{}已下载完毕，无需重新下载",httpFileName);
return;
}
//创建获取下载信息的任务对象
downloadInfoThread = new DownloadInfoThread(contentLength);
//将任务交给线程池执行，每隔一秒执行一次
scheduledExecutorService.scheduleAtFixedRate(downloadInfoThread,1,1, TimeUnit.SECONDS);
//切分对象
List> list = new ArrayList<>();
spilt(url,list);
countDownLatch.await();
//合并文件
if (merge(httpFileName)){
clearTemp(httpFileName);
}
} catch (IOException | InterruptedException e) {
e.printStackTrace();
} finally {
System.out.print("r");
System.out.print("下载完成");
//关闭对象
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
//关闭线程池
scheduledExecutorService.shutdownNow();
poolExecutor.shutdown();
}
}
public void spilt(String url, List> futureList){
try {
//获取下载文件大小
long contentLength = HttpUtils.getHttpFileContentLength(url);
//计算切分后的文件大小
long size = contentLength / Constant.THREAD_NUM;
//计算分块个数
for (int i = 0; i < Constant.THREAD_NUM; i++) {
//计算下载起始位置
long startPos = i * size;
//计算结束位置
long endPos;
if (i == Constant.THREAD_NUM - 1) {
endPos = 0;
}else {
endPos = startPos + size;
}
//如果不是第一块，起始位置+1
if (startPos != 0) {
startPos++;
}
//创建任务
DownloaderTask downloaderTask = new DownloaderTask(url, startPos, endPos, i,countDownLatch);
//提交任务
Future submit = poolExecutor.submit(downloaderTask);
futureList.add(submit);
}
}catch (IOException e){
e.printStackTrace();
}
}
public boolean merge(String fileName){
LogUtils.info("开始合并文件{}",fileName);
byte[] buffer = new byte[Constant.BYTE_SIZE];
int len = -1;
try (RandomAccessFile accessFile = new RandomAccessFile(fileName, "rw")){
for (int i = 0; i < Constant.THREAD_NUM; i++) {
try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(fileName + ".temp" + i));){
while ((len = bis.read(buffer)) != -1) {
accessFile.write(buffer,0,len);
}
}
}
}catch (Exception e){
e.printStackTrace();
return false;
}
return true;
}
public boolean clearTemp(String fileName){
for (int i = 0; i < Constant.THREAD_NUM; i++) {
File file = new File(fileName + ".temp" + i);
file.delete();
}
return true;
}
}
DownloaderTask提交给线程池的任务，也就是分块任务
public class DownloaderTask implements Callable{ private String url; private long startPos; private long endPos; //分块的块号 private int part; private CountDownLatch countDownLatch; public DownloaderTask(String url, long startPos, long endPos, int part, CountDownLatch countDownLatch) { this.url = url; this.startPos = startPos; this.endPos = endPos; this.part = part; this.countDownLatch = countDownLatch; } @Override public Boolean call() throws Exception { //获取文件名 String httpFileName = HttpUtils.getHttpFileName(url); //分块的文件名 httpFileName = httpFileName + ".temp" + part; //下载路径 httpFileName = Constant.PATH + httpFileName; //获取分块下载连接 HttpURLConnection httpURLConnection = HttpUtils.getHttpURLConnection(url, startPos, endPos); try ( InputStream inputStream = httpURLConnection.getInputStream(); BufferedInputStream bis = new BufferedInputStream(inputStream); RandomAccessFile accessFile = new RandomAccessFile(httpFileName, "rw"); ){ byte[] buffer = new byte[Constant.BYTE_SIZE]; int len = -1; //循环读取数据 while ((len = bis.read(buffer)) != -1) { //1秒内下载数据之和 DownloadInfoThread.downSize.add(len); accessFile.write(buffer,0,len); } }catch (FileNotFoundException e){ LogUtils.error("下载文件不存在{}",url); return false; }catch (Exception e){ LogUtils.error("下载出现异常"); return false; }finally { httpURLConnection.disconnect(); countDownLatch.countDown(); } return true; } }
DownloadInfo显示下载信息：
已下载 168.61mb/170.75mb，速度 2320kb/s，剩余时间 0.9s
public class DownloadInfoThread implements Runnable{
//下载文件总大小
private long httpFileContentLength;
//本地已下载文件的大小
public static LongAdder finishedSize = new LongAdder();
//本次累计下载的大小
public static volatile LongAdder downSize = new LongAdder();
//前一次下载的大小
public double prevSize;
public DownloadInfoThread(long httpFileContentLength) {
this.httpFileContentLength = httpFileContentLength;
}
@Override
public void run() {
//计算文件总大小 单位：mb
String httpFileSize = String.format("%.2f",httpFileContentLength / Constant.MB);
//计算每秒下载速度kb
int speed = (int)((downSize.doublevalue() - prevSize) / 1024d);
prevSize = downSize.doublevalue();
//剩余文件的大小
double remainSize = httpFileContentLength - finishedSize.doublevalue() - downSize.doublevalue();
//计算剩余时间
String remainTime = String.format("%.1f", remainSize / 1024d / speed);
if ("Infinity".equalsIgnoreCase(remainTime)) {
remainTime = "-";
}
//已下载大小
String currentFileSize = String.format("%.2f",(downSize.doublevalue() - finishedSize.doublevalue()) / Constant.MB);
String downInfo = String.format("已下载 %smb/%smb，速度 %skb/s，剩余时间 %ss",
currentFileSize,httpFileSize,speed,remainTime);
System.out.print("r");
System.out.print(downInfo);
}
}
常量类，便于修改
public class Constant {
public static final String PATH = "下载文件的存放地址，本地地址";
public static final double MB = 1024d * 1024d;
public static final int BYTE_SIZE = 1024 * 100;
//线程数量
public static final int THREAD_NUM = 5;
}
欢迎分享，转载请注明来源：内存溢出
微信扫一扫
支付宝扫一扫
评论列表（0条）