AmazonS3-Java对接

摘要:
S3是一个全球存储区域网络,它表现为一个超大的硬盘,可以存储AWS用户上传的资源文件。S3可根据AWS用户需求不同创建不同区域、不同名称的Bucket(桶),代码调用的时候可直接根据BucketNamed对Bucket进行访问。
AmazonS3-Java对接

1、Amazons3?

S3 是一个全球存储区域网络(SAN),它表现为一个超大的硬盘,可以存储AWS用户上传的资源文件。S3可根据AWS用户需求不同创建不同区域、不同名称的Bucket(桶),代码调用的时候可直接根据BucketNamed对Bucket进行访问(前提是AWS凭证认证通过)。

2、基础配置

1、AWS用户凭证设置

AWS用户凭证有一下集中方式可配置:
1:LocalFile(本地配置文件)
2:Java.profile/.java文件中设置变量
3:配置环境变量(暂未测试)

1、LoaclFile

1、AWS官网查看用户Id/Key:
AmazonS3-Java对接第1张

AmazonS3-Java对接第2张


2、Windows设置:
指定目录下创建文件夹(.aws)及下属文件(config/credentials)
AmazonS3-Java对接第3张

config文件:

区域可根据自己需求(以及服务所在区域)设置

AmazonS3-Java对接第4张

credentials文件:
AmazonS3-Java对接第5张

3:CLI设置凭证

见:https://www.cnblogs.com/StefanieYang/p/13229382.html

2、pom.xml配置

只需要引入如下依赖即可,版本可根据自行需求设定

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.759</version>
        </dependency>  

3、代码示例

1、AmazonS3连接(正常连接)

packagecom.stefanie.sun.bean.AWS.Rekognition;

importcom.amazonaws.AmazonClientException;
importcom.amazonaws.AmazonServiceException;
importcom.amazonaws.ClientConfiguration;
importcom.amazonaws.Protocol;
importcom.amazonaws.auth.AWSCredentials;
importcom.amazonaws.auth.AWSStaticCredentialsProvider;
importcom.amazonaws.auth.BasicAWSCredentials;
importcom.amazonaws.auth.DefaultAWSCredentialsProviderChain;
importcom.amazonaws.auth.profile.ProfileCredentialsProvider;
importcom.amazonaws.regions.Regions;
importcom.amazonaws.services.s3.AmazonS3;
importcom.amazonaws.services.s3.AmazonS3Client;
importcom.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
importcom.amazonaws.util.IOUtils;
importlombok.extern.slf4j.Slf4j;
importorg.springframework.mock.web.MockMultipartFile;
importorg.springframework.web.multipart.MultipartFile;

importjava.io.File;
importjava.io.FileInputStream;
importjava.io.IOException;
importjava.util.List;

/*** Created with IntelliJ IDEA
 * StefanieSun
 *
 * @Description: AWS S3桶连接
 * @Author: zy
 * @Date: 2020-06-28 15:51
 * @Version: 1.0.0
 */@Slf4j
public classS3client {
  
  private final static String REGION = "us-east-2";
  private final static String ACCESSKEY = "your id";
  private final static String SECREYKEY = "your key";

  public static void main(String[] s) throwsException {
        AmazonS3 s3 =amazonS31();
        try{
            //遍历bucket信息
            for(Bucket bucket : s3.listBuckets()) {
                System.out.println("bucker名称:" + bucket.getName() + "/bucker创建时间:" + bucket.getCreationDate() +
                        "/bucker-owner:" +bucket.getOwner());
            }
        } catch(AmazonClientException ace) {
            log.error(ace.getMessage());
        }
    }

  
  //1-读取默认配置连接AmazonS3
    public staticAmazonS3 amazonS31() {
        AWSCredentials credentials = null;
        try{
            credentials = new ProfileCredentialsProvider("default").getCredentials();
        } catch(Exception e) {
            throw new AmazonClientException("Exception on credentials.", e);
        }
        AmazonS3 s3 = newAmazonS3Client(credentials);
        returns3;
    }

   
  //2-默认连接方式
  public staticAmazonS3 amazonS32() {
      AmazonS3 s3 =AmazonS3ClientBuilder.defaultClient();
      returns3;
  }
  
  
  //3-手动设置区域连接
  public staticAmazonS3 amazonS33(){
   AWSCredentials credentials = null;
        try{
            //获取凭证信息
            credentials = newProfileCredentialsProvider().getCredentials();
        } catch(Exception e) {
            throw newAmazonClientException(
                    "Cannot load the credentials from the credential profiles file. " +
                            "Please make sure that your credentials file is at the correct " +
                            "location (~/.aws/credentials), and is in valid format.",
                    e);
        }

        //通过美国区域设置获取
        AmazonS3 s3 =AmazonS3ClientBuilder.standard()
                .withCredentials(newAWSStaticCredentialsProvider(credentials))
                .withRegion(REGION)
                .build();
        returns3;
  }

  
  //4-使用S3客户端创建连接
  public staticAmazonS3 amazonS34(){
    AmazonS3 s3 = new AmazonS3Client(newDefaultAWSCredentialsProviderChain());
    returns3;
  }

  
  //5-使用Java.profile/.Java变量连接
  public staticAmazonS3 amazonS35(){
     BasicAWSCredentials  awsCreds= newBashicAWSCredentials(ACCESSKEY,SECRETKEY);
     AmazonS3 s3 =AmazonS3ClientBuilder.standard()
             .withCredentials(newAWSStaticCredentialsProvider(awsCreds))
             .withRegion(Region.US-EASE-2)
             .build();
     returns3;
  }
}

2、异常区域无法访问

1、错误信息

场景:在使用 "us-east-1"区域访问的时候无法连接,其他区域均可正常连接

java.lang.reflect.InvocationTargetException: nullat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
    at com.amazonaws.http.conn.$Proxy2.connect(Unknown Source)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:993)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:999)
    at com.stefanie.sun.bean.AWS.Rekognition.S3client.main(S3client.java:49)
Caused by: java.net.UnknownHostException: s3.amazonaws.com
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
    at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    ... 28common frames omitted
10:14:49.175 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-3: Shutdown connection
10:14:49.175 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -Connection discarded
10:14:49.175 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 3][route: {s}->https://s3.amazonaws.com:443][total available: 0; route allocated: 0 of 50; total allocated: 0 of 50]
10:14:49.175 [main] ERROR com.stefanie.sun.bean.AWS.Rekognition.S3client -Unable to execute HTTP request: s3.amazonaws.com

Process finished with exit code 0

2、原因分析

1:需要调用的服务在US-EAST-1区,Bucket也在US-EAST-1区,但是连接时无法访问,提示如上所示为请求有问题
2:通过日志比对访问成功时与访问失败的日志可以看出https://------后面的连接有问题,如下图所示:
AmazonS3-Java对接第6张

由上图可以看出https请求地址不同。

3:在找出时http请求地址有问题的情况下,查看源代码,找出问题,但是并未找到哪里设置有问题。

4:在源码中查出AmazonS3.endpoint属性存储http请求地址,故更改属性即可。

5:使用原有生成AmazonS3的方式无法更改属性(在构建的时候无法更改属性)。

3、解决办法

 public staticAmazonS3 amazonS36() {
        AWSCredentials credentials = newBasicAWSCredentials(ACCESSKEY, SECRETKEY);
        ClientConfiguration clientConfig = newClientConfiguration();
        clientConfig.setProtocol(Protocol.HTTP);
        clientConfig.withSignerOverride("S3SignerType");
        //noinspection deprecation
        AmazonS3 s3 = newAmazonS3Client(credentials, clientConfig);
        s3.setEndpoint("https://s3.us-east-1.amazonaws.com");//访问地址类似,只是区别与用户凭证的验证
        returns3;
    }

4、上传图片

/*** 上传文件
     *
     * @paramfilePath   本地文件路径
     * @parambucketName 桶名称
     * @paramkeyName    桶路径以及名称  实例:"assets/test/1513132153/1.png";
     * @throwsIOException
     */
    private static void uploadFileToBucket(String filePath, String bucketName, String keyName) throwsIOException {
        AmazonS3 s3Client =amazonS36();
        File imageFile = newFile(filePath);
        FileInputStream in = newFileInputStream(imageFile);
        MultipartFile file = new MockMultipartFile("file", imageFile.getName(), "text/plain", IOUtils.toByteArray
                (in));
        ObjectMetadata metadata = newObjectMetadata();
        metadata.setContentType(file.getContentType());
        metadata.setContentLength(file.getSize());
        try{
            s3Client.putObject(newPutObjectRequest(
                    bucketName, keyName, file.getInputStream(), metadata));
        } catch(AmazonServiceException ase) {
            ase.getMessage();
        }
    }

5、查看Bucket中资源

/*** 获取s3桶中数据
     */
    public static voidgetS3Object(String bucketName) {
        AmazonS3 amazonS3 =amazonS36();
        ListObjectsV2Result result =amazonS3.listObjectsV2(bucketName);
        List<S3ObjectSummary> objects =result.getObjectSummaries();
        for(S3ObjectSummary os : objects) {
            System.out.println("* " +os.getKey());
        }
    }

免责声明:文章转载自《AmazonS3-Java对接》仅用于学习参考。如对内容有疑问,请及时联系本站处理。

上篇13、ubuntu终端快捷键(参考 dy9776)vue,element列表大数据卡顿问题,vue列表渲染慢,element表格渲染慢,表格渲染慢(卡),表格全选卡,使用umy-ui下篇

宿迁高防,2C2G15M,22元/月;香港BGP,2C5G5M,25元/月 雨云优惠码:MjYwNzM=

相关文章

leetcode每日一题(2020-07-18):97. 交错字符串

题目描述:给定三个字符串 s1, s2, s3, 验证 s3 是否是由 s1 和 s2 交错组成的。 今日学习:1.遇到字符串,多想想前缀和以及动规2.滚动数组优化:只和当前以及上一状态有关的可以进行空间优化 题解:1.我想的稍微复杂一点的dp2.官方dp3.优化dp /** * @param {string} s1 * @param {string}...

使用 minio 搭建私有对象存储云。aws-php-sdk 操作object

How to use AWS SDK for PHP with Minio Server aws-sdk-php is the official AWS SDK for the PHP programming language. In this recipe we will learn how to use aws-sdk-php with Minio s...

数据对齐

许多计算机系统对数据类型的合法地址做了一些限制,要求某种数据类型对象的地址必须是某个值K(2,4,8)的倍数,这种堆积限制简化形成了处理器和存储器系统之间接口的硬件设计,总之就是为了方便高效的读取数据,于是就有了数据对齐。 struct S1 { char k; double i; char c; }; Sizeof(S1)的...

利用S3fs在Amazon EC2 Linux实例上挂载S3存储桶

一、准备!!! 1、使用拥有足够权限的IAM账号登录AWS控制台 2、创建S3存储桶,给存储桶命名如“my-bucket”(如果使用已有存储桶,本步骤可略过) 3、有该S3存储桶访问权限的 IAM 用户,并为该IAM用户创建访问密钥 关于如何创建IAM用户,请参见:http://docs.aws.amazon.com/zh_cn/IAM/latest/U...

AWS S3 读取流的问题

因为业务,需要将两张MongoDB 表的数据切换到 AWS 的S3中,再切换中发现下面的代码;先删除整个TempFiles 目录,将流写到特定文件中,然后再读出来; 通过尝试修改,认为前人是为了解决AWS 获取的流是不能直接拿来,才这么处理的。但是这样处理会有并发问题。 var request = new GetObjectRequest { Bucke...

aws cli command line interface的安装与使用

安装 在centos中安装aws cli,安装依赖python,先装好python,然后按下述命令执行 yum install wget wget https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install awscli 配置 在使用之前需要配置自己的key ID及 Access K...